Embeddings for word sense disambiguation: an evaluation study

Recent years have seen a dramatic growth in the popularity of word embeddings mainly owing to their ability to capture semantic information from massive amounts of textual content. As a result, many tasks in Natural Language Processing have tried to take advantage of the potential of these distributional models. In this work, we study how word embeddings can be used in Word Sense Disambiguation, one of the oldest tasks in Natural Language Processing and Artificial Intelligence. We propose different methods through which word embeddings can be leveraged in a state-of-the-art supervised WSD system architecture, and perform a deep analysis of how different parameters affect performance. We show how a WSD system that makes use of word embeddings alone, if designed properly, can provide significant performance improvement over a state-of-the-art WSD system that incorporates several standard WSD features

SensEmbed: Learning sense embeddings for word and relational similarity

Word embeddings have recently gained considerable popularity for modeling words in different Natural Language Processing (NLP) tasks including semantic similarity measurement. However, notwithstanding their success, word embeddings are by their very nature unable to capture polysemy, as different meanings of a word are conflated into a single representation. In addition, their learning process usually relies on massive corpora only, preventing them from taking advantage of structured knowledge. We address both issues by proposing a multifaceted approach that transforms word embeddings to the sense level and leverages knowledge from a large semantic network for effective semantic similarity measurement. We evaluate our approach on word similarity and relational similarity frameworks, reporting state-of-the-art performance on multiple datasets

Embedding Words and Senses Together via Joint Knowledge-Enhanced Training

Word embeddings are widely used in Nat-ural Language Processing, mainly due totheir success in capturing semantic infor-mation from massive corpora. However,their creation process does not allow thedifferent meanings of a word to be auto-matically separated, as it conflates theminto a single vector. We address this issueby proposing a new model which learnsword and sense embeddings jointly. Ourmodel exploits large corpora and knowl-edge from semantic networks in order toproduce a unified vector space of wordand sense embeddings. We evaluate themain features of our approach both qual-itatively and quantitatively in a variety oftasks, highlighting the advantages of theproposed method in comparison to state-of-the-art word- and sense-based models

Neural-grounded semantic representations and word sense disambiguation: a mutually beneficial relationship

Language, in both the written and the oral forms, is the ground basis of living in society. The same basic kinds of rules and representations are shared across all the languages. Understand those rules is the objective of Natural Language Processing (NLP), the computerized discipline responsible to analyze and generate language. Building complex computational systems that mimic the human language and are capable to interact and collaborate with us is the holy grail of Natural Language Processing. Semantic representations are the rock-solid foundation on which many successful applications of NLP depend. Their main purpose is to extract and highlight the most important semantic features of textual data. Whereas over the years different approaches have been presented, lately, embeddings have become the dominant paradigm on vectorial representation of items. Currently, many outstanding NLP tasks rely on embeddings to achieve their performance. Embeddings are semantic spaces that carry valuable syntactic and semantic information. The name groups a set of feature learning techniques based on neural networks. Concretely, these techniques are capable to learn semantic spaces that effectively represent words as low-dimensional continuous vectors. They also maintain the structure of language by representing diverse lexical and semantic relations by a relation-specific vector offset. With the increasing amount of available text, as well as the increased computing power, techniques which take advantage of large volumes of unstructured data, as word embeddings, have become the prevailing approach of semantic representation of natural language. However, despite their enormous success, common word-embeddings approaches came with two inherent flaws: these representations are incapable to handle ambiguity, as senses of polysemous words are aggregated into single vectors. In addition, most word embeddings rely only on statistical information of word occurrences, leaving aside existing rich knowledge of structured data. To tackle the problem of polysemy, a fundamental task of Natural Language Processing (NLP), Word Sense Disambiguation (WSD), seems particularly suitable. The task, an open problem in the discipline, aims at identifying the correct meaning of word based given its context. Concretely, it links each word occurrence to a sense from a predefined inventory. Most successful approaches for WSD combine the use of unstructured data, manually annotated datasets and semantic resources. In the present thesis we address the issue of of ambiguity in semantic representations from a multimodal perspective. Firstly, we introduce and investigate new neural-based approaches to build better word and sense embeddings relying on both statistical data and prior semantic knowledge. We employ diverse techniques of WSD for linking word occurrences to their correct meaning on large amounts of raw corpora. Then, we use the resulting data as training input for learning the embeddings. We show the quality of these representations by evaluating them on standard semantic similarity frameworks reporting state-of-the-art performance on multiple datasets. Secondly, we show how these representations are capable to create better WSD systems. We introduce a new way to leverage word representations which outperforms current WSD approaches in both supervised and unsupervised configurations. We show that our WSD framework, based solely on embeddings, is capable to surpass WSD approaches based on standard features. Thirdly, we propose two new technique for leveraging semantic-annotated data. We incorporate more semantic features resulting in an increment in the performance compared with our initial approaches. We close the loop by showing that our semantic representations enhanced with WSD are also suitable for improving the task of WSD itself

LSTMEmbed: learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories

While word embeddings are now a de facto standard representation of words in most NLP tasks, recently the attention has been shifting towards vector representations which capture the different meanings, i.e., senses, of words. In this paper we explore the capabilities of a bidirectional LSTM model to learn representations of word senses from semantically annotated corpora. We show that the utilization of an architecture that is aware of word order, like an LSTM, enables us to create better representations. We assess our proposed model on various standard benchmarks for evaluating semantic representations, reaching state-of-the-art performance on the SemEval-2014 word-to-sense similarity task. We release the code and the resulting word and sense embeddings at http://lcl.uniroma1.it/LSTMEmbed